SUBMITTED TO IEEE TRANSACTIONS ON SIGNAL PROCESSING, DECEMBER, 2009. 



1 



Distributed Learning in Multi- Armed Bandit with 

Multiple Players 

Keqin Liu, Qing Zhao 
University of California, Davis, CA 95616 
{kqliu, qzhao}@ucdavis.edu 

Abstract 

We formulate and study a decentralized multi-armed bandit (MAB) problem. There are M distributed 
players competing for N independent arms. Each arm, when played, offers i.i.d. reward according to a 
distribution with an unknown parameter. At each time, each player chooses one arm to play without ex- 
changing observations or any information with other players. Players choosing the same arm collide, and, 
depending on the collision model, either no one receives reward or the colliding players share the reward in 
an arbitrary way. We show that the minimum system regret of the decentralized MAB grows with time at the 
same logarithmic order as in the centralized counterpart where players act collectively as a single entity by 
exchanging observations and making decisions jointly. A decentralized policy is constructed to achieve this 
optimal order while ensuring fairness among players and without assuming any pre-agreement or information 
exchange among players. Based on a Time Division Fair Sharing (TDFS) of the M best arms, the proposed 
policy is constructed and its order optimality is proven under a general reward model. Furthermore, the 
basic structure of the TDFS policy can be used with any order-optimal single-player policy to achieve order 
optimality in the decentralized setting. We also establish a lower bound on the system regret growth rate for 
a general class of decentralized polices, to which the proposed policy belongs. This problem finds potential 
applications in cognitive radio networks, multi-channel communication systems, multi-agent systems, web 
search and advertising, and social networks. 

Index Terms 

Decentralized multi-armed bandit, system regret, distributed learning, cognitive radio, web search and 
advertising, multi-agent systems. 
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I. Introduction 

A. The Classic MAB with A Single Player 

In the classic multi-armed bandit (MAB) problem, there are N independent arms and a single player. 
Playing arm i (i = 1, • • • ,N) yields i.i.d. random rewards with a distribution parameterized by an unknown 
Oi. At each time, the player chooses one arm to play, aiming to maximize the total expected reward in the long 
run. Had the reward model of each arm been known to the player, the player would have always chosen the 
arm that offers the maximum expected reward. Under an unknown reward model, the essence of the problem 
lies in the well-known tradeoff between exploitation (aiming at gaining immediate reward by choosing the 
arm that is suggested to be the best by past observations) and exploration (aiming at learning the unknown 
reward model of all arms to minimize the mistake of choosing an inferior arm in the future). 

Under a non-Bayesian formulation where the unknown parameters = (Q\, ■ ■ ■ , #jv) are considered deter- 
ministic, a commonly used performance measure of an arm selection policy is the so-called regret or the cost 
of learning defined as the reward loss with respect to the case with known reward models. Since the best 
arm can never be perfectly identified from finite observations (except certain trivial scenarios), the player can 
never stop learning and will always make mistakes. Consequently, the regret of any policy grows with time. 

An interesting question posed by Lai and Robbins in 1985 CD is on the minimum rate at which the regret 
grows with time. They showed in HI that the minimum regret grows at a logarithmic order under certain 
regularity conditions. The best leading constant was also obtained, and an optimal policy was constructed 
under a general reward model to achieve the minimum regret growth rate (both the logarithmic order and the 
best leading constant). In 1987, Anantharam et al. extended Lai and Robbins's results to MAB with multiple 
plays: exactly M (M < N) arms can be played simultaneously at each time |2]. They showed that allowing 
multiple plays changes only the leading constant but not the logarithmic order of the regret growth rate. They 
also extended Lai-Robbins policy to achieve the optimal regret growth rate under multiple plays. 

B. Decentralized MAB with Distributed Multiple Players 

In this paper, we formulate and study a decentralized version of the classic MAB, where we consider M 
(M < N) distributed players. At each time, a player chooses one arm to play based on its local observation 
and decision history. Players do not exchange information on their decisions and observations. Collisions occur 
when multiple players choose the same arm, and, depending on the collision model, either no one receives 
reward or the colliding players share the reward in an arbitrary way. The objective is to design distributed 
policies for each player in order to minimize, under any unknown parameter set 0, the rate at which the system 
regret grows with time. Here the system regret is defined as the reward loss with respect to the maximum 
system reward obtained under a known reward model and with centralized scheduling of the players. 
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The single -player MAB with multiple plays considered in O is equivalent to a centralized MAB with 
multiple players. If all M players can exchange observations and make decisions jointly, they act collectively 
as a single player who has the freedom of choosing M arms simultaneously. As a direct consequence, the 
logarithmic order of the minimum regret growth rate established in [2 ] provides a lower bound on the minimum 
regret growth rate in a decentralized MAB where players cannot exchange observations and must make 
decisions independently based on their individual local observation!]. 

C. Main Results 

In this paper, we show that in a decentralized MAB where players can only learn from their individual 
observations and collisions are bound to happen, the system can achieve the same logarithmic order of the 
regret growth rate as in the centralized case. Furthermore, we show that this optimal order can be achieved 
under a fairness constraint that requires all players accrue reward at the same rate. 

A decentralized policy is constructed to achieve the optimal order under the fairness constraint. The proposed 
policy is based on a Time Division Fair Sharing (TDFS) of the M best arms where no pre-agreement on 
the time sharing schedule is needed. It is constructed and its order optimality is proven under a general 
reward model. The TDFS decentralized policy thus applies to and maintains its order optimality in a wide 
range of potential applications as detailed in Sec. II-DI Furthermore, the basic structure of TDFS is not tied 
with a specific single -player policy. It can be used with any single -player policy to achieve an efficient and 
fair sharing of the M best arms among M distributed players. More specifically, if the single-player policy 
achieves the optimal logarithmic order in the centralized setting, then the corresponding TDFS policy achieves 
the optimal logarithmic order in the decentralized setting. The order optimality of the TDFS policy is also 
preserved when player's local polices are built upon different order-optimal single -player polices. 

We also establish a lower bound on the leading constant in the regret growth rate for a general class of 
decentralized polices, to which the proposed TDFS policy belongs. This lower bound is tighter than the trivial 
bound provided by the centralized MAB considered in 0, which indicates, as one would have expected, that 
decentralized MAB is likely to incur a larger leading constant than its centralized counterpart. 

D. Applications 

With its general reward model, the TDFS policy finds a wide area of potential applications. We give a few 
examples below. 

Consider first a cognitive radio network where secondary users independently search for idle channels 
temporarily unused by primary users O. Assume that the primary system adopts a slotted transmission 

'While intuitive, this equivalence requires the condition that the M best arms have nonnegative means (see Sec. |IV| for details). 
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structure and the state (busy/idle) of each channel can be modeled by an i.i.d. Bernoulli process. At the 
beginning of each slot, multiple distributed secondary users need to decide which channel to sense (and 
subsequently transmit if the chosen channel is idle) without knowing the channel occupancy statistics {i.e., 
the mean Oi of the Bernoulli process). An idle channel offers one unit of reward. When multiple secondary 
users choose the same channel, none or only one receives reward depending on whether carrier sensing is 
implemented. 

Another potential application is opportunistic transmission over wireless fading channels H. In each slot, 
each user senses the fading realization of a selected channel and chooses its transmission power or date rate 
accordingly. The reward can model energy efficiency (for fixed-rate transmission) or throughput. The objective 
is to design distributed channel selection policies under unknown fading statistics. 

Consider next a multi-agent system, where each agent is assigned to collect targets among multiple locations 
(for example, ore mining). When multiple agents choose the same location, they share the reward according 
to a certain rule. The log-Gaussian distribution with an unknown mean may be used as the reward model 
when the target is fish [5] or ore 151 . 

Another potential application is Internet advertising where multiple competing products select Web sites to 
posts advertisements. The reward obtained from the selected Web site is measured by the number of interested 
viewers whose distribution can be modeled as Poisson Q. 

E. Related Work 

In the context of the classic MAB, there have been several attempts at developing index-type policies that are 
simpler than Lai-Robbins policy by using a single sample mean based statistic [8,9]. However, such a simpler 
form of the policy was obtained at the price of a more restrictive reward model and/or a larger leading constant 
in the logarithmic order. In particular, the index policy proposed in [8] mainly targets at several members 
of the exponential family of reward distributions. Auer et al. proposed in [9] several index-type policies that 
achieve order optimality for reward distributions with a known finite support. 

Under a Bernoulli reward model in the context of cognitive radio, MAB with multiple players was considered 
in iflOl and ifTTl . In iflOl . a heuristic policy based on histogram estimation of the unknown parameters was 
proposed. This policy provides a linear order of the system regret rate, thus cannot achieve the maximum 
average reward. In ifTTTl . Anandkumar et al. have independently established order-optimal distributed policies 
by extending the index-type single-user policies proposed in O. Compared to [11], the TDFS policy proposed 
here applies to more general reward models (for example, Gaussian and Poisson reward distributions that have 
infinite support). It thus has a wider range of potential applications as discussed in Sec. II-Di Furthermore, the 
policies proposed in iTTTTl are specific to the single -player polices proposed in O, whereas the TDFS policy can 
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be used with any order-optimal single-player policy to achieve order optimality in the decentralized setting. 
Another difference between the policies proposed in [11] and the TDFS policy is on user fairness. The policies 
in ifTTI orthogonalize users into different channels that offer different throughput, whereas the TDFS policy 
ensures that each player achieves the same time-average reward at the same rate. One policy given in IfTTI 
does offer probabilistic fairness in the sense that all users have the same chance of settling in the best channel. 
However, given that the policy operates over an infinite horizon and the order optimality is asymptotic, each 
user only sees one realization in its lifetime, leading to different throughput among users. A lower bound on 
the achievable growth rate of the system regret is also given in ifTTTl . which is identical to the bound developed 
here. The derivation of the lower bound in this work, however, applies to a more general class of policies 
and, first given in [12], precedes that in IfTTI . In terms of using collision history to orthogonalizing players 
without pre-agreement, the basic idea used in this work is similar to that in QX]. The difference is that in 
the TDFS policy, the players are orthogonalized via settling at different offsets in their time sharing schedule, 
while in [11], players are orthogonalized to different channels. Recently, results obtained in IfTTI have been 
extended to incorporate unknown number of users for Bernoulli reward model in the context of cognitive 
radio lfl3l . 

A variation of centralized MAB in the context of cognitive radio has been considered in |[T4l where a 
channel offers independent Bernoulli rewards with different (unknown) means for different users. This more 
general model captures contention among secondary users that are in the communication range of different sets 
of primary users. A centralized policy that assumes full information exchange and cooperation among users 
is proposed which achieves the logarithmic order of the regret growth rate. We point out that the logarithmic 
order of the TDFS policy is preserved in the decentralized setting when we allow players to experience 
different reward distributions on the same arm provided that players have the common set of the M best arms 
and each of the M best arms has the same mean across players. It would be interesting to investigate whether 
the same holds when players experience different means on each arm. 

F. Notations 

Let \A\ denote the cardinality of set A. For two sets A and B, let A\B denote the set consisting of all 
elements in A that do not belong to B. For two positive integers k and I, define kCdl = ((k— 1) mod l)+l, which 
is an integer taking values from 1,2, •• • ,1. Let Prg{-} denote the probability measure when the unknown 
parameter in the associated distribution equals to 9. 
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II. Classic Results on Single-Player MAB 

In this section, we give a brief review of the main results established in HI, |2), JSJ, O on the classic 
MAB with a single player. 

A. System Regret 

Consider an iV-arm bandit with a single player. At each time t, the player chooses exactly M (1 < M < N) 
arms to play. Playing arm i yields i.i.d. random reward Si(t) drawn from a univariate density function f(s; 6i) 
parameterized by 9{. The parameter set = (9\, ■ ■ ■ , 9n) is unknown to the player. Let n(9i) denote the mean 
of Si(t) under the density function f(s;9i). Let 1(9,9') = Eg [log[ jD^& ]] be the Kullback-Liebler distance 
that measures the dissimilarity between two distributions parameterized by 9 and 9', respectively. 

An arm selection policy it = {7r(t)}^ 1 is a series of functions, where ir(t) maps the previous observations 
of rewards to the current action that specifies the set of M arms to play at time t. The system performance 
under policy 7r is measured by the system regret i?J(0) defined as the expected total reward loss up to time 
T under policy 7r compared to the ideal scenario that is known to the player (thus the M best arms are 
played at each time). Let ex be a permutation of {1, • • • , N} such that Li(9 a n\) > fi(9 a ( 2 )) > • • • > A*(0<r(iV))- 
We have 

where S^ffity) is the random reward obtained at time t under action Tr(t), and K w [-] denotes the expectation 
with respect to policy it. The objective is to minimize the rate at which Rt(&) grows with T under any 
parameter set by choosing an optimal policy it*. 

We point out that the system regret rate is a finer performance measure than the long-term average reward. 
All policies with a sublinear regret rate would achieve the maximum long-term average reward. However, the 
difference in their performance measured by the total expected reward accrued over a time horizon of length 
T can be arbitrarily large as T increases. It is thus of great interest to characterize the minimum regret rate 
and construct policies optimal under this finer performance measure. 

A policy is called uniformly good if for any parameter set 0, we have R^(Q) = o(T h ) for any b > 0. Note 
that a uniformly good policy implies the sub-linear growth of the system regret and achieves the maximum 
long-term average reward Sj l £ 1 /x(0 o .( J )) which is the same as in the case with perfect knowledge of 0. 

B. The Logarithmic Order and the Optimal Policy 

We present in the theorem below the result established in [1,2] on the logarithmic order as well as the 
leading constant of the minimum regret growth rate of the single -player MAB. 
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Theorem [1,2]: Under the regularity conditions (conditions C1-C4 in Appendix A), we have, for any 
uniformly good policy ir, 

lhn f i^(e) KQ*m) - Mi) 

Lai and Robbins also constructed a policy that achieves the lower bound on the regret growth rate given 
in £[]) under single play (M = 1) HI (which was extended by Anantharam et al. to M > 1 in O). Under this 
policy, two statistics of the past observations are maintained for each arm. Referred to as the point estimate, 
the first statistic is an estimate of the mean fi(9i) given by a function h k of the past observations on this arm 
(k denotes the total number of observations). The second statistic is the so-called confidence upper bound 
which represents the potential of an arm: the less frequently an arm is played, the less confident we are about 
the point estimate, and the higher the potential of this arm. The confidence upper bound, denoted by g t ^, thus 
depends on not only the number k of observations on the arm but also the current time t in order to measure 
how frequently this arm has been played. 

Based on these two statistics, Lai-Robbins policy operates as follows. At each time t, among all "well- 
sampled" arms, the one with the largest point estimate is selected as the leader denoted as l t . The player then 
chooses between the leader l t and a round-robin candidate r t = t N to play. The leader l t is played if 
and only if its point estimate exceeds the confidence upper bound of the round-robin candidate r t . A detailed 
implementation of this policy is given in Fig. [T] 

Lai and Robbins ID have shown that for point estimates h k and confidence upper bounds g t> k satisfying 
certain conditions (condition C5 in Appendix A), the above policy is optimal, i.e., it achieves the minimum 
regret growth rate given in £[]). For Gaussian, Bernoulli, and Poisson reward models, h k and g t: k satisfying 
condition C5 are given below (H. 

h k = (2) 

g t)k = inf{A : A > h k and I(h k , A) > 1 ° g(t fc ~ 1) }. (3) 

While g tj k given in © is not in closed-form, it does not affect the implementation of the policy. Specifically, 
the comparison between the point estimate of the leader l t and the confidence upper bound of the round-robin 
candidate r t (see Fig. CD is shown to be equivalent to the following two conditions HI : 

T r t ,t 

Consequently, we only require the point estimate hf. to implement the policy. 
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Lai-Robbins Policy for Single-Player MAB |1| 

• Notations and Inputs: let r n < denote the number of times that arm n has been played up to 
(but excluding) time t and S n ,i> • • • , Sn,r n t tne P ast observations obtained from arm n. Fix 
5 € (0, 1/N). 

• Initializations: in the first N steps, play each arm once. 

• At time t (t > N), among all arms that have been played at least (t — 1)5 times, let l t 
denote the arm with the largest point estimate (referred to as the leader): 

It = arg max h Tn t (S n i, ■■■ , S HtTn J. 

Let r t = t N be. the round robin candidate at time t. The player plays the leader l t 

if h nw>t (Si t)1 , ■ ■ ■ ,S hiTlut ) > gt,T rtJ {S rt ,i, ■ ■ ■ ,Sr t , Trt J and the round-robin candidate r t 
otherwise. 

Fig. 1. Lai-Robbins policy for single-player MAB (T|. 



C. Order-Optimal Index Policies 

Since Lai and Robbins's seminal work, researchers have developed several index-type policies that are 
simpler than Lai-Robbins policy by using a single sample mean based statistic [8,9]. Specifically, under such 
an index policy, each arm is assigned with an index that is a function of its sample mean and the arm with 
the greatest index will be played in each slot. To obtain an initial index, each arm is played once in the first 
N slots. 

The indexes proposed in j8l for Gaussian, Bernoulli, Poisson and exponential distributed reward models are 
given in (@]). Except for Gaussian distribution, this index policy only achieves the optimal logarithmic order 
of the regret growth rate but not the best leading constant- 

' x + (2 logft-l)+2^1oglog(t-l) )1/2) Gaussian 

x + min{(2 log(t ~ 1)+ ^ 1 ° slog(f ' 1) ) 1 /2/2, 1}, Bernoulli 

x + min{(2a log(t ~ 1)+ ; n 1 ° glog(f ~ 1) ) 1 /2/2, a}, Poisson ' ^ 
x + 6mm{(2 log( ^ 1)+ ^ 1 ° glog(f ~ 1) ) 1 /2 i i}, Exponential 

where !(•) is the index function, x is the sample mean of the arm, and a, b are upper bounds of all possible 



I(x) 



2 The optimal indexes for Bernoulli, Poisson and exponential distributions are also developed in [8]. However, the indexes are not 
given in closed-form and are difficult to implement. 
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values of 9 in the Poisson and exponential reward models, respectively. 

Based on [8], a simpler order-optimal sample mean based index policy was established in [9] for reward 
distributions with a known finite support: 

!(*)=* + (2 l ^-V) V*. (5) 

T n ,t 

III. Decentralized MAB: Problem Formulation 

In this section, we formulate the decentralized MAB with M distributed players. In addition to conditions 
C1-C5 required by the centralized MAB, we assume that the M best arms have distinct nonnegative mean|J 

In the decentralized setting, players may collide and may not receive reward that the arm can potentially offer. 
We thus refer to Si(t) as the state of arm i at time t (for example, the busy/idle state of a communication 
channel in the context of cognitive radio). At time t, player i (1 < i < M) chooses an action ai{t) 6 
{1, • • • , N} that specifies the arm to play and observes its state S a .^(t). The action is based on the 
player's local observation and decision history. Note that the local observation history of each player also 
includes the collision history. As shown in Sec. |V-C[ the observation of a collision is used to adjust the local 
offset of a player's time sharing schedule of the M best arms to avoid excessive future collisions. 

We define a local policy 7Tj for player i as a sequence of functions 7Tj = {iTi(t)}t>i, where 7Tj(t) maps 
player i's past observations and decisions to action a,i(t) at time t. The decentralized policy it is thus given 
by the concatenation of the local polices for all players: 

Define reward Y(t) as the total reward accrued by all players at time t, which depends on the system collision 
model as given below. 

Collision model 1: When multiple players choose the same arm to play, they share the reward in an arbitrary 
way. Since how players share the reward has no effect on the total system regret, without loss of generality, 
we assume that only one of the colliding players obtains a random reward given by the current state of the 
arm. Under this model, we have 

F(i) = Ef =1 I i (t)5,(t), 

where Ij(t) is the indicator function that equals to 1 if arm j is played by at least one player, and otherwise. 

Collision model 2: When multiple players choose the same arm to play, no one obtains reward. Under this 
model, we have 

Y(t) = ^f =1 tj(t)Sj(t), 
3 This condition can be relaxed to the case that a tie occurs at the Mth largest mean. 
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where Ij (t) is the indicator function that equals to 1 if arm j is played by exactly one player, and otherwise. 
The system regret of policy 7r is thus given by 

Note that the system regret in the decentralized MAB is defined with respect to the same best possible reward 
TT>jL 1 Li(O a ^) as in its centralized counterpart. The objective is to minimize the rate at which Rt(@) grows 
with time T under any parameter set by choosing an optimal decentralized policy. Similarly, we say a 
decentralized policy is uniformly good if for any parameter set 0, we have Rt(Q) = o(T b ) for any b > 0. 
To address the optimal order of the regret, it is sufficient to focus on uniformly good decentralized polices 
provided that such policies exist. 

We point out that all results developed in this work apply to a more general observation model. Specifically, 
the arm state observed by different players can be drawn from different distributions, as long as players have 
the common set of the M best arms and each of the M best arms has the same mean across players. This 
relaxation in the observation model is particularly important in the application of opportunistic transmission 
in fading channels, where different users experience different fading environments in the same channel. 

IV. The Optimal Order of The System Regret 

In this section, we show that the optimal order of the system regret growth rate of the decentralized MAB 
is logarithmic, the same as its centralized counterpart as given in Sec. ITT] 

Theorem 1: Under both collision models, the optimal order of the system regret growth rate of the decen- 
tralized MAB is logarithmic, i.e., for an optimal decentralized policy n*, we have 

L(Q) < lim inf < lim sup ^L^l < [/( ) (6) 

T^oo logT r^oo logT 

for some constants L(0) and U(Q) that depend on 0. 

Proof: The proof consists of two parts. First, we prove that the lower bound for the centralized MAB 
given in CO) is also a lower bound for the decentralzied MAB. Second, we construct a decentralized policy 
(see Sec. [V) that achieves the logarithmic order of the regret growth rate. 

It appears to be intuitive that the lower bound for the centralized MAB provides a lower bound for the 
decentralized MAB. This, however, may not hold when some of the M best arms have negative means 
(modeling the punishment for playing certain arms). The reason is that the centralized MAB considered in O 
requires exactly M arms to be played at each time, while in the decentralized setting, fewer than M arms may 
be played at each time when players choose the same arm. To obtain an lower bound for the decentralized 
MAB, we need to consider a centralized MAB where the player has the freedom of playing up to M arms at 
each time. When the player knows that all arms have nonnegative means, it is straightforward to see that the 
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two conditions of playing exactly M arms and playing no more than M arms are equivalent. Without such 
knowledge, however, this statement needs to be proven. 

Lemma 1: Under the condition that the M best arms have nonnegative means, the centralized MAB that 
requires exactly M arms are played at each time is equivalent to the one that requires at most M arms are 
played at each time. 

Proof: See Appendix B. ■ 

V. An Order-Optimal Decentralized Policy 

In this section, we construct a decentralized policy ir* F that achieves the optimal logarithmic order of the 
system regret growth rate under the fairness constraint. 

A. Basic Structure of the Decentralized TDFS Policy tt f 

The basic structure of the proposed policy tt* f is a time division structure at each player for selecting the 
M best arms. For the ease of the presentation, we first assume that there is a pre-agrement among players 
so that they use different phases (offsets) in their time division schedule. For example, the offset in each 
player's time division schedule can be predetermined based on the player's ID. In Sec. IV-CI we show that 
this pre-agreement can be eliminated while maintaining the order-optimality and fairness of the TDFS policy, 
which leads to a complete decentralization among players. 

Consider, for example, the case of M = 2. The time sequence is divided into two disjoint subsequences, 
where the first subsequence consists of all odd slots and the second one consists of all even slots. The pre- 
agreement is such that player 1 targets at the best arm during the first subsequence and the second best arm 
during the second subsequence, and player 2 does the opposite. 

Without loss of generality, consider player 1. In the first subsequence, player 1 applies a single -player 
policy, say Lai-Robbins policy, to efficiently learn and select the best arm. In the second subsequence, the 
second best arm is learned and identified by removing the arm that is considered to be the best and applying 
the Lai-Robbins policy to identify the best arm in the remaining N — 1 arms (which is the second best among 
all N arms). Note that since the best arm cannot be perfectly identified, which arm is considered as the 
best in an odd slot t (in the first subsequence) is a random variable determined by the realization of past 
observations. We thus partition the second subsequence into multiple mini-sequences depending on which arm 
was considered to be the best in the preceding odd slot and thus should be removed from consideration when 
identifying the second best. Specifically, as illustrated in Fig. |2j the second subsequence is divided into N 
disjoint mini-sequences, where the ith mini-sequence consists of all slots that follow a slot in which arm i was 
played (i.e., arm i was considered the best arm in the preceding slot that belongs to the first subsequence). 
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In the ith mini-sequence, the player applies Lai-Robbins policy to arms {1, • • • ,i — 1, i + 1, ••• ,N} after 
removing arm i. 

In summary, the local policy for each player consists of N + 1 parallel Lai-Robbins procedures: one is 
applied in the subsequence that targets at the best arm and the rest N are applied in the N mini-sequences 
that target at the second best arm. These parallel procedures, however, are coupled through the common 
observation history since in each slot, regardless of which subsequence or mini-sequence it belongs to, all the 
past observations are used in the decision making. We point out that making each mini-sequence use only 
its own observations is sufficient for the order optimality of the TDFS policy and simplifies the optimality 
analysis. However, we expect that using all available observations leads to a better constant as demonstrated 
by simulation results in Sec. IVIII 
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t = 1 2 3 4 5 6 7 8 

The first subsequence (targeting at the best arm) 

The first mini-sequence (targeting at the second best arm when arm 1 is considered as the best) 
The second mini-sequence (targeting at the second best arm when arm 2 is considered as the best) 
The third mini-sequence (targeting at the second best arm when arm 3 is considered as the best) 



Fig. 2. The structure of player l's local policy under n* F (M — 2, N = 3. Let a(l) denote the arm considered as the best by player 
1 in the first subsequence. In this example, player 1 divides the second subsequence (i.e., all the even slots) into three mini-sequences, 
each associated with a subset of 2 arms after removing the arm considered to be the best in the first subsequence.). 



The basic structure of the TDFS policy ir F is the same for the general case of M > 2. Specifically, the time 
sequence is divided into M subsequences, in which each player targets at the M best arms in a round-robin 
fashion with a different offset. Suppose that player 1 has offset 0, i.e., it targets at the /cth (1 < k < M) 
best arm in the fcth subsequence. To player 1, the /cth (k > 1) subsequence is then divided into mini- 
sequences, each associated with a subset of N — k + 1 arms after removing those k— 1 arms that are considered 
to have a higher rank than k. In each of the C^_ x mini-sequences, Lai-Robbins single-player policy is applied 
to the subset of N — k + 1 arms associated with this mini-sequence. Note that while the subsequences are 
deterministic, each mini-sequence is random. Specifically, for a given slot t in the kth subsequence, the mini- 
sequence which it belongs to is determined by the specific actions of the player in the previous k— 1 slots. For 
example, if arms 1, • • ■ , k — 1 are played in the previous k — 1 slots, then slot t belongs to the mini-sequence 



SUBMITTED TO IEEE TRANSACTIONS ON SIGNAL PROCESSING, DECEMBER, 2009. 



13 



associated with the arm set of {k, • • • , N}. A detailed implementation of the proposed policy for a general 
M is given in Fig. [3] 

Note that the basic structure of the TDFS policy ir* F is general. It can be used with any single-player policy 
to achieve an efficient and fair sharing of the M best arms among M distributed players. Furthermore, players 
are not restricted to using the same single -player policy. Details are given in Sec. IV-BI 

B. Order-Optimality under Fairness Constraint 

Compared to the single -player counterpart given in [1,2], the difficulties in establishing the logarithmic regret 
growth rate of -k* f are twofold. First, compared to the centralized problem where a single player observes 
M different arms simultaneously, each player here can only observe one arm at each time. Each player is 
thus learning the entire rank of the M best arms with fewer observations. Furthermore, since the rank of 
any arm can never be perfectly identified, the mistakes in identifying the ith (1 < i < M) best arm will 
propagate into the learning process for identifying the (i + l)th, up to the Afth best arm. Second, without 
centralized scheduling, collisions are bound to happen even with pre-agreed offset on sharing the M best 
arms since players do not always agree on the rank of the arms. Such issues need to be carefully dealt with 
in establishing the logarithmic order of the regret growth rate. 

Theorem 2: Under the TDFS policy ir F , we have, for some constant C(G), 

lim S u P 5-^<C(6). (7) 

T^oo log T 



Let 



Under collision model 1, 

C(G) = M{Y l fL l x k Li{e a{ i ) ) - S n:M (g„) <M( e ) j7K—jr , /■ 

1 \yni V<t(M)) 

Under collision model 2, 

C(G) = M&ti^ixMe^) - X n . Aen)<Ke „ (m)( ,(e n ) max{- — - \ 0}). 

Proof: Note that the system regret is given by the sum of the regrets in the M subsequences. By symmetry 
in the structure of ir F , all subsequences experience the same regret. Consequently, the system regret is equal 
to M times the regret in each subsequence. We thus focus on a particular subsequence and show that the 
regret growth rate in this subsequence is at most logarithmic. Without loss of generality, consider the first 
subsequence in which the ith player targets at the ith (1 < i < M) best arm. 

To show the logarithmic order of the regret growth rate, The basic approach is to show that the number of 
slots in which the ith player does not play the ith best arm is at most logarithmic with time. This is done by 
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The Decentralized TDFS Policy ir* F 

Without loss of generality, consider player i. 

• Notations and Inputs: In addition to the notations and inputs of Lai-Robbins single -player 
policy (see Fig. [TJ), let mk(t) denote the number of slots in the kth subsequence up to (and 
including) t, and let mj^{t) denote the number of slots in the mini-sequence associated 
with arm set M up to (and including) t. 

• At time t, player i does the following. 

1. If t belongs to the ith subsequence (i.e., t M = i), player i targets at the best arm 
by carrying out the following procedure. 

If rrii(t) < N, play arm mj(t). Otherwise, the player chooses between a leader 
l t and a round-robin candidate r t = rrii(t) N, where the leader l t is the arm 
with the largest point estimate among all arms that have been played for at least 
(rrii(t) — 1)5 times. The player plays the leader l t if its point estimate is larger than the 
confidence upper bound of r t . Otherwise, the player plays the round-robin candidate r t . 

2. If t belongs to the fcth (k / i) subsequence (i.e., t M = k), the player targets at the 
jth best arm where j = (k — i + M + 1)0 M by carrying out the following procedure. 
If mk(t) < N, play arm mk(t). Otherwise, the player targets at the jth best arm. Let 
At denote the set of j — 1 arms played in the previous j — 1 slots. Slot t thus belongs to 
the mini-sequence associated with the subset Mt = {1, • • • , N}\At of arms. The player 
chooses between a leader and a round-robin candidate defined within Mt- Specifically, 
among all arms that have been played for at least (mj^ t (t) — 1)5 times, let l t denote 
the leader. Let r t = m^ t (t) (N — j + 1) be the round-robin candidate where, for 
simplicity, we have assumed that arms in Mt are indexed by 1, • • • , N — j + 1. The 
player plays the leader l t if its point estimate is larger than the confidence upper bound 
of r t . Otherwise, the player plays the round-robin candidate r t . 



Fig. 3. The decentralized TDFS policy Tip. 
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establishing in Lemma [3] a lower bound on the number of slots in which the ith player plays the zth best arm 
in the first subsequence. 

To show Lemma |3j we first establish Lemma |2] by focusing on the dominant mini-sequence of the first 
subsequence. Without loss of generality, consider player i. Define the dominant mini-sequence as the one 
associated with arm set {<r(i), • • • , o~(N)} (i.e., the i — 1 best arms are correctly identified and removed in this 
mini-sequence). Lemma [2] shows that, in the dominant mini-sequence, the number of slots in which player i 
does not play the ith best arm is at most logarithmic with time. 

Lemma 2: Let tP t denote the number of slots up to time T in which arm n is played in the dominant 
mini-sequence associated with arm set ■ ■ ■ ,a(N)}. Then, for any arm n with n(9 n ) < n{O a n\), we 

have, 

E K T ] i 

hmsup - — '— < r. (8) 

t^oo logT I(9 n ,e a( i)) 

Proof: Note that this lemma is an extension of Theorem 3 in |T]. The proof of this lemma is more 

complicated since the mini-sequence is random and the decisions made in this mini-sequence depend on all 

past observations (no matter to which mini-sequence they belong). See Appendix C for details. ■ 

Next, we establish Lemma [3] The basic approach is to show that the length of the dominant mini-sequence 
dominates the lengths of all other mini-sequences in the first subsequences. Specifically, we show that the 
number of slots that do not belong to the dominant mini-sequence is at most logarithmic with time. This, 
together with Lemma |2] that characterizes the dominant mini-sequence, leads to Lemma [3] below. 

Lemma 3: Let t^uxx denote the number of slots in which player i plays the ith best arm in the first 
subsequence up to time T, we have 

hm sup — < Xi , (9) 

t^oo log T 

where 

Xi = E ^ lS ^ )< ^« )) /(^^))' (10) 
Proof: The proof is based on an induction argument on i. See Appendix D for details. ■ 

From Lemma |3l for all 1 < i < M, the number of slots that the ith best arm is not played by player i 

is at most logarithmic with time. Consequently, for all j ^ i, the number of slots that player j plays the ith 

best arm is also at most logarithmic with time, i.e., the number of collisions on the zth best arm is at most 

logarithmic with time. Since a reward loss on the ith (1 < i < M) best arm can only happen when it is not 

played or a collision happens, the reward loss on the ith best is at most logarithmic order of time, leading to 

the logarithmic order of the regret growth rate. 
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To establish the upper bound C(0) on the constant of the logarithmic order of the system regret growth 
rate, we consider the worst-case collisions on each arm. See Appendix E for details. ■ 

From Theorem Q] and Theorem [2j the decentralized policy is order-optimal. Furthermore, as given in 
Theorem [3] below, the decentralized policy ensures fairness among players under a fair collision model that 
offers all colliding players the same expected reward. For example, if only one colliding player can receive 
reward, then all players have equal probability to be lucky. 

Theorem 3: Define the local regret for player % under tt f as 

where Yi(t) is the immediate reward obtained by player i at time t. Under a fair collision model, we have, 

Theorem [3] follows directly from the symmetry among players under w F . It shows that each player achieves 
the same time average reward jjTHji ll fj,(9 a .(j^) at the same rate. We point out that without knowing the 
reward rate that each arm can offer, ensuring fairness requires that each player identify the entire set of the 
M best arms and share each of these M arms evenly with other players. As a consequence, each player needs 
to learn which of the possibilities is the correct choice. This is in stark difference from polices that make 
each player target at a single arm with a specific rank (for example, the ith player targets solely at the ith 
best arm). In this case, each player only needs to distinguish one arm (with a specific rank) from the rest. 
The uncertainty facing each player, consequently, the amount of learning required, is reduced from to N. 
Unfortunately, fairness among players is lost. 

As mentioned in Sec. IV-A1 the proposed policy irp can be used with any single -player policy, which can 
also be different for different players. More importantly, the order optimality of the TDFS policy is preserved 
as long as each player's single -player policy achieves the optimal logarithmic order in the single player setting. 
This statement can be proven along the same line as Theorem [2] by establishing results similar to Lemma [2] 
and Lemma [3] 



C. Eliminating the Pre-Agreement 

In this subsection, we show that a pre-agreement among players on the offsets of the time division schedule 
can be eliminated in the TDFS policy irp while maintaining its order-optimality and fairness. 

Specifically, when a player joins the system, it randomly generates a local offset uniformly drawn from 
{0, 1, • • • , M — 1} and plays one round of the M arms considered to be the best. For example, if the random 
offset is 1, the player targets at the second, the third, • • • , the Mth, and then the best arms in the M subsequent 
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slots, respectively. If no collision is observed in this round of M plays, the player keeps the same offset; 
otherwise the player randomly generates a new offset. Over the sequence of slots where the same offset is 
used, the player implements the local policy of ir* F (given in Fig. [3]) with this offset. To summarize, each 
player implements M parallel local procedures of ir F corresponding to M different offsets. These parallel 
procedures are coupled through the observation history, i.e., each player uses its entire local observation history 
in learning no matter which offset is being used. 

Note that players can join the system at different time. We also allow each player to leave the system for 
an arbitrary finite number of slots. 

Theorem 4: The decentralized TDFS policy -k f without pre-agreement is order-optimal and fair. 

Proof: See Appendix F. ■ 

VI. A Lower Bound for a Class of Decentralized Polices 

In this section, we establish a lower bound on the growth rate of the system regret for a general class 
of decentralized policies, to which the proposed policy ir* F belongs. This lower bound provides a tighter 
performance benchmark compared to the one defined by the centralized MAB. The definition of this class of 
decentralized polices is given below. 

Definition 1: Time Division Selection Policies The class of time division selection (TDS) policies consists 
of all decentralized polices it = [tti, ■ ■ ■ ,7Tm] that satisfy the following property: under local policy 7r,, there 
exists < dij < 1 (1 < i,j < M, Tjj^aij = 1 V i) independent of the parameter set such that the 
expected number of times that player i plays the jth (1 < j < M) best arm up to time T is aijT — o(T b ) 
for all b > 0. 

A policy in the TDS class essentially allows a player to efficiently select each of the M best arms according 
to a fixed time portion that does not depend on the parameter set 6. It is easy to see that the TDFS policy 
ir F (with or without pre-agreement) belongs to the TDS class with ciij = 1/M for all 1 < i, j < M. 

Theorem 5: For any uniformly good decentralized policy it in the TDS class, we have 

W to£F - s i=i s ;^)<M<^>) 1{9j ,9 a(i) ) ■ (12) 

Proof: The basic approach is to establish a lower bound on the number of slots in which each player 
plays an arm that does not belong to the M best arms. By considering the best case that they do not collide, 
we arrive at the lower bound on the regret growth rate given in (fT2l) . The proof is based on the following 
lemma, which generalizes Theorem 2 in HI. To simplify the presentation, we assume that the means of all 
arms are distinct. However, Theorem [5] and Lemma 0] apply without this assumption. 
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Lemma 4: Consider a local policy 7T$. If for any parameter set and b > 0, there exists a ©-independent 
positive increasing function v(T) satisfying v (T) — > oo as T — > oo such that 

E[v(T)-r a{ ^ T ]<o((v(T)) b ), (13) 

then we have, V k > j, 

Hminf g^ 1 (14) 

t^oo logu(T) " I(6 a{k): e a{j) ) 

Note that Lemma [4] is more general than Theorem 2 in [1] that assumes v(T) = T and j = 1. The proof 
of Lemma |4] is given in Appendix G. 

Consider a uniformly good decentralized policy it in the TDS class. There exists a player, say player u\, 
that plays the best arm for at least T/M — o(T b ) times. Since at these times, u\ cannot play other arms, there 
must exist another player, say player 112, that plays the second best arm for at least T/(M(M — 1)) — o(T b ) 
times. It thus follows that there exist M different players ui, • • ■ , um such that under any parameter set 0, 
the expected time player itj (1 < i < M) plays the ith best arm is at least r/JJ*- =1 (M — j + 1) — o{T b ) 
times. Based on Lemma 0] for any arm j with /u(#j) < m(^o-(JW))j the expected time that player Ui plays arm 
j is at least 1/1(9 j\ 9 a ^) log(T/ n}=i(^ — * + 1)) — °0°g By considering the best case that players do 
not collide, we arrive at (fl2l ). 

■ 

VII. Simulation Examples 

In this section, we study the performance (i.e., the leading constant of the logarithmic order) of the 
decentralized TDFS policy in different applications through simulation examples. 

A. Cognitive Radio Networks: Bernoulli Reward Model 

We first consider a Bernoulli reward model using cognitive radio as an example of application. There are 
M secondary users independently search for idle channels among N channels. In each slot, the busy/idle state 
of each channel is drawn from Bernoulli distribution with unknown mean, i.e., f(s;6) = 8 S (1 — for 
s G {0, 1}. When multiple secondary users choose the same channel for transmission, they will collide and 
no one gets reward (collision model 2). 

In Fig. @1 we plot the performance of -k* f (with pre-agreement) using different single-player policies. The 
leading constant of the logarithmic order is plotted as a function of N with fixed M. From Fig. |U we 
observe that adopting the optimal Lai-Robbins single-player policy in ir* F achieves the best performance. This 
is expected given that Lai-Robbins policy achieves the best leading constant in the single-player case. We 
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-1 W The TDFS policy based on Lai-Robbins policy (w/o coupling) 



: i i i i i : 

3 4 5 6 7 8 9 

Number of Channels (N) 

Fig. 4. The performance of 7i> built upon different single-player policies (Bernoulli distributions, global horizon length T = 
5000, M = 2, 6 = [0.1, 0.2, • ■ • , 

also observe that coupling the parallel single-player procedures through the common observation history in 
7Tp leads to a better performance compared to the one without coupling. 

In Fig. [U we study the impact of eliminating pre-agreement on the performance of it p. We plot the leading 
constant of the logarithmic order as a function of M with fixed N. We observe that eliminating pre-agreement 
comes with a price in performance in this case. We also observe that the system performance degrades as 
M increases. One potential cause is the fairness property of the policy that requires each player to learn the 
entire rank of the M best arms. 

B. Multichannel Communications under Unknown Fading: Exponential Reward Model 

In this example, we consider opportunistic transmission over wireless fading channels with unknown fading 
statistics. In each slot, each user senses the fading condition of a selected channel and then transmits data 
with a fixed power. The reward obtained from a chosen channel is measured by its capacity (maximum data 
transmission rate) C = log(l + SNR). We consider the Rayleigh fading channel model where the SNR of each 
channel is exponential distributed with an unknown mean. When multiple users choose the same channel, no 
one succeeds (collision model 2). Note that a channel that has a higher expected SNR also offers a higher 
expected channel capacity. It is thus equivalent to consider SNR as the reward which is exponential distributed. 
Specifically, we have f(s;8) = 1/9 exp(— x/ff) for s E TZ + . 
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Fig. 5. The performance of n* F based on Lai-Robbins policy (Bernoulli distributions, global horizon length T = 5000, N = 9, 
[0.1, 0.2, ■•■ , 0.9]). 



In Fig. [6l we plot the leading constant of the logarithmic order as a function of N with fixed M. We 
observe that in this example, eliminating pre-agreement has little impact on the performance of the TDFS 
policy. 

C. Target Collecting in Multi-agent Systems: Gaussian Reward Model 

In this example, we consider the Gaussian reward model arisen in the application of multi-agent systems. 
At each time, M agents independently select one of N locations to collect targets {e.g., fishing or ore mining). 
The reward at each location is determined by the fish size or ore quality, which has been shown in [5, 6] to 
fit with log-Gaussian distribution. The reward at each location has an unknown mean but a known variance 
which is the same across all locations. When multiple agents choose the same location, they share the reward 
(collision model 1). 

Note that the log-Gaussian and Gaussian reward distributions are equivalent since the arm ranks under these 
two distributions are the same when the arms have the same variance. We can thus focus on the Gaussian 
reward model, i.e., f(s;9) = (2ira 2 )- 1 / 2 exp{-(s - 6) 2 /(2a 2 )} for s £ K. 

In Fig. |7J we plot i?^, F /log(T) as a function of global horizon T. We observe that the regret growth rate 
quickly converges as time goes, which implies that the policy can achieve a strong performance within a short 
finite period. In Fig. [8j we plot the leading constant of the logarithmic order as a function of M with fixed 
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Fig. 6. The performance of 7i> based on Agrawal's index policy (Exponential distributions, global horizon length T = 5000, M = 
2, 6 = [1, 2, ... , N]), 

N. Similar to the previous example, eliminating pre-agreement has little impact on the performance of the 
TDFS policy. 

VIII. Conclusion 

In this paper, we have studied a decentralized formulation of the classic multi-armed bandit problem by 
considering multiple distributed players. We have shown that the optimal system regret in the decentralized 
MAB grows at the same logarithmic order as that in the centralized MAB considered in the classic work by 
Lai and Robbins HI and Anantharam, et al. Q. A decentralized policy that achieves the optimal logarithmic 
order has been constructed. A lower bound on the leading constant of the logarithmic order is established 
for polices in the TDS class, to which the proposed policy belongs. Future work includes establishing tighter 
lower bound on the leading constant of the logarithmic order and investigating decentralized MAB where 
each arm has different means to different players. 

Appendix A. 

Let @ denote the set of all possible values of the unknown parameter 0. 
Regularity Conditions: 

CI Existence of mean: li{6) exists for any G@. 
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Fig. 7. The convergence of the regret growth rate under np based on Lai-Robbins policy (Gaussian distributions, a = 1, M = 



4, N = 9, 9=[1, 2, ■•■ , 9]). 

C2 Positive distance: < I{9,9') < oo whenever li(8) < ll(9'). 

C3 Continuity of I(6,6'):Ve > and li(9) < li(9'),35 > such that \I(8, 9')-I(9, 9")\ < e whenever fj,(9') < 
H(6") < KG') + S. 

C4 Denseness of ©: MO and 5 > 0, 39' G 6 such that li(8) < ll(9') < fi{9) + 5. 
Conditions on point estimate and confidence upper bound: 

C5 For any 9 G@, we have 

(i) PT {g t , k (S(l), ■■■ , S(k)) > r for all k < t} = 1 - o(t _1 ) for every r < ll(9); 

(ii) lim 4 o(limsup^ 00 S*- 1 1 Pr e { 5tifc (5(l), • • • ,S(k)) > fi(9') - e}/logt) < 1/1(9,9') 
whenever ll(9') > fi{9); 

(iii) 9t,k is nondecreasing in t for every fixed k = 1, 2, • • ■ ; 

(iv) g t ,k >h k Vt; 

(v) Pre{max S t<k<t \h k - > = o^ 1 ) V e > 0, < 5 < 1. 



Under the condition that the M best arms have nonnegative means, the maximum expected immediate 



Appendix B. Proof of Lemma [T] 



reward obtained under the ideal scenario that 6 is known is given by E| 



^(^cr(j))- Under a uniformly good 
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Fig. 8. The performance of n* F based on Lai-Robbins policy (Gaussian distributions, a = 1, global horizon length T = 5000, N 

io, e= [i, 2, •■• , io]). 



policy 7T, we have, for any arm n with fi(9 n ) > /i(#o-(M))> 

E[r n , T ] = T-o(T b ), V b> 0. 
Following Theorem 3.1 in [2], the above equation implies that for any arm j with < /^(^ ct (m)), 

limM gP%j>_J 

The regret growth rate is thus lower bounded by 

. .i?r(e) . v n(0 <M) ) - ufa) 

fe^ W ^ E i^XM^ ,) i(^,^ (M) ) ' 

Since the optimal policy that chooses exactly M arms at each time achieves the above lower bound [1,2], it 

is also optimal for the MAB that chooses up to M arms at each time. The two problems are thus equivalent. 

Appendix C. Proof of Lemma[2] 

Let It denote the leader among the arm set {cr(i), • • • ,a(N)}. Consider arm n with Li(6 n ) < /u(#<j(i)). Let 
T>(T) denote the set of slots in the dominant mini-sequence up to time T. 

For any e € (0,/x(f? CT (j)) — let iVi(T) denote the number of slots in T>{T) at which arm n is 

played when the leader is the ith best arm and the difference between its point estimate and true mean does 
not exceed e, N 2 (T) the number of slots in V(T) at which arm n is played when the leader is the ith best 
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arm and the difference between its point estimate and true mean exceeds e, and Ns(T) the number of slots 
in V(T) when the leader is not the zth best arm. Recall that each arm is played once in the first N slots. We 
have 

r^ T < l + iVi(T) + iV 2 (T)+iV3(T). (15) 

Next, we show that E[Ni(T)], E[N 2 (T)], and E[N 3 (T)] are all at most in the order of logT. 
Consider first E[N\(T)]. Based on the structure of Lai-Robbins single -player policy, we have 

iVi(T) = |{1< t < T : t £ {V(T)}, fi(6 lt ) = n{6 a(i) ), \h Th t - n{6 a(i) )\ < e and arm n is played at time t}\ 

< 1 + |{1 < j < T - 1 : g aJ (S n i, ■■• , S nj ) > ^{6 a{€) ) - e for some j<s< t}\ 

< l + \{l<j<T-l:g T!j (S nl ,---,S nj )>fx(6 a{i) )-e}\. (16) 

Under condition C5, for any p > 0, we can choose e sufficiently small such that 

E(|{1 < j < T - 1 : ffrj(S nl , ■ ■ ■ , S nj ) > fi(9 a{i) ) - e }|) 

lim sup — — — 

r^oo logT 

^J=i PT{9Tj{S nl , • • • , S nj ) > p(9 a{i) ) - e } 

= hm sup ■ 



< 



1+p 



I{6n->0o{i)) 



Thus, 



EINUT)} 1 
limsup^-^< (17) 

Consider E[N 2 (T)]. Define c D (t) = \{s : s <t,s,t G V{T)}\ as the number of slots in V(T) that are no 
larger than t. Since the number of observations obtained from l t is at least (co(t) — 1)5, under condition C5, 
we have, 

Pr{c D (t) = s for some s < t,p(9 h ) = n{9 a{j) ), \h Tlt t - n{9 a(i) )\ > e} 

< Pr{ sup {hjiS^,--- ,S ltd )-p(9 lt )\>e} 

j>S(s-l) 

= o^s- 1 ). (18) 

We thus have 

E[N 2 (T)} = E(|{Kt<T: t G V(T), p(6 h ) = p(9 a{i) ), \K h , t - v(9 a{i) )\ > e}\) 

< Sf =1 Pr{c D (t) = s for some s < t < T,p(6 h ) = n(9 a ^), \h Tlt t - n(9 a ^)\ > e} 

= o(logT). (19) 
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Next, we show that E[iV 3 ] = o(logT). 

Choose < ei < (/i(6? CT (j)) — /x(0 o -( i+1 )))/2 and c > (1 — Nd)^ 1 . For r = 0, 1, • • • , define the following 
events. 

A r = ni< fc <iv{ max \h s (S u / k \ 1 ,---,S u f k \ s )-Lt(6 a f k \)\<ei}, 

B r = {g Sm ,j(S a (i) t i, ■ • • , S a ^j) > n(9 a (iy) - e for all 1 < j < 5m, c r ~ l <m< c r+1 , and s m > m}. 

By (TT8T ). we have Pr(^4 r ) = o(c _r ). Consider the following event: 

C r = {g m +i,j(S a{i)A ,- ■ ■ , > M( e <j(i)) - e i for all 1 < z < 5m, and c r_1 < m < c r+1 }. (20) 

Under condition C5, it is easy to see that B r D C r . From Lemma 1 — (i) in [1], Pr(C%) = o(c~ r ). We thus 
have Pr(B r ) = o{c~ r ). 

Consider time t. Define the event D r = {t G T>(T), c r_1 < co(t) — 1 < c r+1 }. When the round-robin 
candidate r t = cr(i), we show that on the event A r n B r n -D r , <r(i) must be played. It is sufficient to focus on 
the nontrivial case that /i(#/ f ) < M^o-fi))- Since r; tji > (cD(i)-l)<5, on ^nD,., we have h Tlf t < pb{6 a u\) — e\. 
We also have, on A r n B r n -D r , 

5^(0,* (5^)^, ■ • ■ , 5<T(i) ir „ w ,t) > M(0<r(i)) (21) 

Arm cr(i) is thus played on A r n -B r H D r . Since (1 — c~ 1 )/N > 5, for any c r < s < c r+1 , there exists 
an vq such that on A r Pi -B r , Tcr(i),i > (V-^0( s ~~ c r_1 ~~ 2JV) > (5s for all r > tq. It thus follows that on 
A r n S r PI D r , for any c r < co(i) — 1 < c r+1 , we have > (co(i) — 1)5, and cr(«) is thus the leader. 

We have, for all r > tq, 

Pr(Z t ^ o-(i),t G D(T),c r < c D (t) - 1 < c r+1 ) < Pr(A r n D r ) + Pr(B r n D r ) = o(c" r ). (22) 
Therefore, 

E[JV 3 (T)] = E[\{t > 1 : t G D(T), V, < M^(i))}0 

< Sjjp 1 Pr(c (i) - 1 = s for some t> s,l t ^ a{i),t G £>(T)) 

< l + sS T1 S c ,< s < c , +1 Pr(3t, ^<7(i),tGD(T) )Cfl (t)-l = S ) 
= ft^o(l) 

= o(logT). (23) 
From (fTTT ). (fl9l ), (1231 . we arrive at Lemma [2] 
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Appendix D. Proof of Lemma [3] 

We prove by induction on i (1 < i < M). Consider % = 1. In this case, Lai-Robbins policy is applied to all 
arms by player 1. Let E[f^y T ] denote the expected total number of times in the first sequence that player i 
does not play the ith best arm up to time T. From Lemma [2j we have 

l T snp wt = s ^M0»XK^ci>) l T snp "wt" - Xl ' (24) 
Since {T ~^ M) < W^fi? + Mi),r] < jj, we have 

hm sup — = hm sup — — — - < x x , (25) 

T->oo logT T ^oo logT 

which shows that © holds for i = 1. Let Efr^] denote the number of slots in the dominant mini-sequence 



of player i up to time T. From ©, we have, for i = 1, 

T/M - E[xf 4 



1+1 ] 

hmsup — — ^ < xi. (26) 

T^oo log T 

Assume © and (l26l ) hold for z = A;. To complete the induction process, consider i = k + 1. Let E[r^ T ] 
denote the expected number of slots that the ith best arm is played by player i in its dominant mini-sequence 
up to time T, and Efr-^ij^,] the expected number of slots that the ith best arm is not played by player i in 
its dominant mini-sequence up to time T. Based on Lemma |2j we have 

Efr* } , 

hmsup — — — < S ( fln)< (e )— — -. (27) 



Since E[r^] = E[r^ T + r% )>T ] < E[t^ t + ^ 0>T ], we have 

T/M - E% ii)jT ] T/M-E[t% ] 

lim sup < Iim sup ■ 



T->oo logT r^oo logT 

T/M-E[t^]+E[t^ t ] 
= lim sup 1 — 

T^oo log T 

< Xk+i- (28) 

We thus proved Lemma [3] by induction. 

Appendix E. Proof of the Upper Bound C(0) in Theorem[2] 

Collision Model 1. Consider the first subsequence. Since the expected number of slots that the ith best arm is 
played by player i dominates that by other players, the expected total reward E[Yj(T)] obtained from the ith 
best arm by all players up to time T satisfies 

hmsup : — < XiH{e a u)). (29) 

T^oo log T 



SUBMITTED TO IEEE TRANSACTIONS ON SIGNAL PROCESSING, DECEMBER, 2009. 



27 



For an arm n with Lt(O n ) < n(0 a ^ M ^), the worst case is that player M plays arm n for l/I(9 n ,9 a (M)) logT — 
o(logT) times up to time T (by Lemma|2]). From (1291 ). the expected total reward S^ x E[li(T)] obtained from 
all arms and players up to time T satisfies 

T^tL^{e u{i) )/M - Y.t.miT)] 

hm sup — 

T^oo log T 

< Efi^i^e^o) - S Tt : At (g n )< /x (e o . (M) ) J(a a sK^n)- (30) 

1 {"n, Vcr(M)) 

The expected total reward E^E^E^T)] obtained from all arms and players over all the M subsequences 
up to time T thus satisfies 

ortiM ..In 



lim sup 

< M(Ef£ 1 x^(6> CT( ^ ) ) - E w:/l(0ti)<At( e ) * r^W)- (31) 

Collision Model 2. Consider the first subsequence. By Lemma [3l the expected number of slots in which the 
zth best arm is played by player j (j ^ i) that is at most Xj log T + o(log T) up to time T. Together with ©, 
the expected total reward E[1^(T)] obtained from arm i by all players up to time T satisfies 

hmsup — < T lk=1 x k Li(e a{i) . (32) 

T^oo log T 

For an arm n with n{9 n ) < /^(#<j(m))> th e worst case is that player i plays arm n for 1/I(9 n , 9 a i{\) logT — 
o(log T) times up to time T. Combined with the worst case collisions, the expected reward obtained on arm 
n is at least (max{l/I(0 n , 9 M ) - ^fi^ 1 l/I(9 n , 6 a{i) ), O}/x(0 n )) log T - o(logT) up to time T. From (fJU), 
the expected total reward E^ 1 E[Fj(T)] obtained from all arms and players up to time T satisfies 

rE^ lM (0 (r(i) )/M-E^ 1 E[y i (r)] 



lim sup 

T^oo log T 

< E-e"^^)-^^^ (33) 

The expected total reward Ef =1 Y,f =1 E[Yi(T)} obtained from all arms and players over all the M subsequences 
up to time T thus satisfies 



rE^ l/ x(0 (7(i) )-E^ 1 E^ 1 E[F i (T)] 



lim sup ■ 

t^oo log T 

< M^fi^x^^) - E„ :M9n)<M9g(M)) max{ 1 - S^ 1 1 , 0}/x(6> w )). (34) 

Appendix F. Proof of Theorem H] 

Based on the structure of 7r^ without pre-agreement, each player's local policy consists of disjoint rounds, 
each of which consists of M slots where M arms are played with a particular local offset. We say that a 
round is in group o (0 < o < M — 1) if it corresponds to local offset o. We also say that a round is normal 
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if the player plays the M best arms in the correct order according to its local offset, otherwise, the round 
is singular. Let t denote the global reference time. We say that slot t is normal if every player's round that 
contains slot t is normal, otherwise, slot t is singular. 

Note that the set of slots where a reward loss incurs is a subset of the singular slots and normal slots 
involving collisions. To prove the logarithmic order of the system regret growth rate, it is thus sufficient to 
show the following: (i) the expected number of singular slots is at most logarithmic with time; (ii) the expected 
number of normal slots involving collisions is at most logarithmic with time. 

To show (i), we consider an arbitrary player. Similar to the proof of Theorem |2 it can be shown that in 
any group o, the expected number of singular rounds is at most logarithmic with time. Summed over all M 
groups, the total expected number of singular rounds is at most logarithmic with time. Since that a singular 
round corresponds to M singular slots, the expected number of singular slots is at most logarithmic with time. 

To show (ii), we first show that between any two consecutive singular slots, the expected number of normal 
slots involving collisions is uniformly bounded by some constant (independent of the two singular slots being 
considered). Note that a collision occurs in a normal slot only when multiple players have the same global 
offset (with respect to the global time t). Since a player will randomly generate a new local offset if a collision 
is observed, it is easy to show that, between any two consecutive singular slots, the expected number of normal 
slots needed for all players to settle at different global offsets is uniformly bounded. Consequently, between any 
two consecutive singular slots, the expected number of normal slots involving collisions is uniformly bounded. 
It thus follows that the expected number of normal slots involving collisions is at the same logarithmic order 
as the expected number of singular slots. 

The fairness of tt* f without pre-agreement follows directly from the symmetry among players. 

Appendix G. Proof of Lemma |4] 
For any 5 > 0, under condition C1-C4, we can choose a parameter A such that 



Consider the new parameter set 0' where the mean of the kth best arm (say arm /) is replaced by A. Since 
arm / is the jth best arm under 0', from ( fT3T ), we have, for any < b < 5, 



and 



(35) 



E e ,[v(T)-n, T ] <o((v(T)) b ), 



(36) 
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where ti t is the number of times that arm I is played up to time T. There thus exists a random function 
c(T) with c(T) + v(T) - t 1>t > a.s. and E e >[c(T)] > such thafl 

E e , [c(T) + v(T) - n )T ] = o((v(T)) b ). (37) 

We have 

n? r trr\ j_ rrr\ i w r , ^ i (1 - (5) log(«(T)) (1 - g) log(«(T)) 
Ee' c(-T) +v{T) - n,T\ = Ee>[c(T) +v{T) - ti,t\ti,t < -rTa \1 J Pr< 9' \ Tl , T < T\ / 

+E e 4c(T) + u(T) - t; !T |t; !T > — \Pie>{ T l,T > ^ rr } 

■> tit r ( r r\ j- ! r r\ I ^ (1 - S)log(u(r)) 1n _ (1 - tf) logfoCT)) , 

> E ,[c(T)+v{T) - Ti >T \Ti tT < J77, tt JPre'{">7,T < jtt, rr } 

> (E fl .[c(T)] +v(T) - g < " -.g^'f » }. (38. 

where the fist inequality is due to the fact that c(T) + u(T) — ti t > almost surely. 
Let 5^1, S 1 ^, • • • denote independent observations from arm I. Define 

Aw , J{Sl,i,0o{k)), 



L t = El =1 log 



f(Si,i;X) ' 
and event 

J V.f(T(fc) > A J 

Following (f3Tb and d38l ). we have 

PMC) < Mv < <i^») 



< 



(E e , [c(T)] + «(T) - (1 - 5) log«T))/I(0 CT(fc) , A)) 
o((«(T)) b ) 



(«(T)-(l-*)log(t;(T))/(^ (fc) ,A)) 
= o((v(T)) b - 1 ), (39) 

where the last inequality is due to the fact that Ee'[c(T)] > 0. 

We write C as the union of mutually exclusive events C s = {t^t = s, L s < (1 — b)logv(T)} for each 
integer s < (1 - 5)log(v(T))/ I(9 a{k) , X). Note that 



IMC} = / dPr e , 

J{t,, t =s, L 3 <(l-b)logv(T)} 



s 



{r iiT = S , L,<(l-6)logl!(T)}" JWl,i,V<r(k)) 

> exp(-(l-6)logu(r))Pr {C s }. (40) 

4 For example, c(T) can be constructed as c(T) = b(T) - v(T) + ti.t, where b(T) = [E e '[u(T) - r,, T ]] + . 
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Based on ( 1391 and (1401 ). we have 

Pr {C} = (log^T))) 1 -^^} 0. (41) 

By the strong law of large numbers, L t /t — > I(9 a ^,X) > as t — > oo a.s. under Pre. This leads to 
maxj<£ Lift — > I(9 a f k \, A) > a.s. under Pre. Since 1 — b > 1 — 5, we have 

lim Pr e {Li >{l-b) log v(T) for some i < ^ ~fj l ° g ^ } = 0, (42) 

T^oo I(0 a{k) ,X) 



which leads to 



lim Prg{n, T < (1 i^ffl . V > (1 " 6) log «(T)} = 0. (43) 
Based on (|4~TI ) and (l43l . we have 

T l lm Pr„K T <<l^lM}=0. (44) 

Based on (I35T ). we arrive at 

lim Pr e {r,, T < } = for any 5 > 0, (45) 

which leads to (fT4l . 
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